I found a bug when using the function geemap.download_ee_image.
Before the proposed fix, if the input image has zero values, they will be used to updat the mask of the input image, which looks to be undesirable. In addition, calling unmask before calling clip does not look to make sense.
I did not tested the proposed fix extensively but I'm providing an example with Google Dynamic World. I had problem trying to save images of this dataset since it has valid zero values.

ee_image = ee.Image("GOOGLE/DYNAMICWORLD/V1/20220119T185709_20220119T185710_T10SFG")
classification = ee_image.select("label")

m = geemap.Map()

VIS_PALETTE = [
    '419bdf',
    '397d49',
    '88b053',
    '7a87c6',
    'e49635',
    'dfc35a',
    'c4281b',
    'a59b8f',
    'b39fe1',
]
vis_params = {'min': 0, 'max': 8, 'palette':VIS_PALETTE}

m.addLayer(classification, vis_params, "DW")
m.center_object(classification, 9)
m

This is the correct output:

Trying to export this image with the current code. Here I set unmask_value to 255, since 0 is a valid class in Google Dynamic World. I also use the parameter set_nodata to avoid geedim to set 0 as the ndata value (it's the default option for uint8 dtype).

geemap.download_ee_image(
    classification,
    filename="ee_image.tif",
    scale=50,
    crs="EPSG:4326",
    region=classification.geometry(),
    unmask_value=255,
    set_nodata=False,
)

The output of this code is a tif file in which the zero class become equal to 255 and the areas outside the polygons used to clip the image become 0. With the proposed the change, the exported image looks correct.

Below I'm providing a code to plot the images.

def plot(file):
    with rasterio.open(file) as src:
        data = src.read(1)

    VIS_PALETTE = [
        '#419bdf',
        '#397d49',
        '#88b053',
        '#7a87c6',
        '#e49635',
        '#dfc35a',
        '#c4281b',
        '#a59b8f',
        '#b39fe1',
    ]
    colors = [mcolors.hex2color(color) for color in VIS_PALETTE]
    cmap = mcolors.ListedColormap(colors)
    data = data.astype(np.float32)
    data[data==255] = np.nan
    plt.imshow(data, cmap=cmap)
    plt.show()

The output images:
Original image (before fix)
plot('ee_image.tif')

Image with proposed fix
plot('ee_image.tif')